Neural Network-Based Face Detection



Henry A. Rowley

har@cs.cmu.edu Shumeet Baluja

baluja@cs.cmu.edu

School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213, USA

Takeo Kanade

tk@cs.cmu.edu

Appears in Computer Vision and Pattern Recognition, 1996.

Abstract

We present a neural network-based face detection system.

A retinally connected neural network examines small win-

dows of an image, and decides whether each window con-

tains a face. The system arbitrates between multiple net-

works to improve performance over a single network. We

use a bootstrap algorithm for training the networks, which

adds false detections into the training set as training pro-

gresses. This eliminates the difﬁcult task of manually se-

lecting non-face training examples, which must be chosen

to span the entire space of non-face images. Compar-

isons with other state-of-the-art face detection systems are

presented; our system has better performance in terms of

detection and false-positive rates.

1 Introduction

In this paper, we present a neural network-based algorithm

to detect frontal views of faces in gray-scale images1.The

algorithms and training methods are general, and can be

applied to other views of faces, as well as to similar object

and pattern recognitionproblems.

Training a neural network for the face detection task

is challenging because of the difﬁculty in characterizing

prototypical“non-face” images. Unlikeface recognition,in

which the classes to be discriminated are different faces, the

twoclassestobediscriminatedinfacedetection are “images

containing faces” and “images not containing faces”. It is

easy to get a representative sample of images which contain

faces, but it is much harder to get a representative sample



This work was partially supportedby a grantfrom Siemens Corporate

Research,Inc., and by the Departmentofthe Army, Army Research Ofﬁce

under grant number DAAH04-94-G-0006. This work was started while

ShumeetBalujawassupportedbyaNationalScienceFoundationGraduate

Fellowship. He is currently supported by a graduate student fellowship

from the NationalAeronauticsand Space Administration, administered by

the Lyndon B. Johnson Space Center. The conclusions in this document

are those of the authors, and do not necessarily represent the policies of

the sponsoring agencies.

1A demonstration at http://www.cs.cmu.edu/˜har/faces.html allows

anyoneto submit images for processing by the face detector, and displays

the detection results for pictures submitted by others.

of those which do not. The size of the training set for the

second class can grow very quickly.

We avoid the problem of using a huge training set for

non-faces by selectively adding images to the training set

as training progresses [Sung and Poggio, 1994]. Detailed

descriptions of this training method, along with the net-

work architecture are given in Section 2. In Section 3, the

performance of the system is examined. We ﬁnd that the

system is able to detect 90.5% of the faces over a test set of

130 images, with an acceptable number of false positives.

Section 4 compares this system with similar systems. Con-

clusions and directions for future research are presented in

Section 5.

2 Description of the system

Our system operates in two stages: it ﬁrst applies a set of

neural network-based ﬁlters toan image, and then arbitrates

the ﬁlter outputs. The ﬁlters examine each location in the

image at several scales, looking for locations that might

contain a face. The arbitrator then merges detections from

individual ﬁlters and eliminates overlapping detections.

2.1 Stage one: A neural network-based ﬁlter

The ﬁrst component of our system is a ﬁlter that receives

as input a 20x20 pixel region of the image, and generates

an output ranging from 1 to -1, signifying the presence or

absence of a face, respectively. To detect faces anywhere in

the input, the ﬁlter is applied at every location in the image.

To detect faces larger than the window size, the input image

is repeatedly subsampled by a factor of 1.2, and the ﬁlter is

applied at each scale.

The ﬁltering algorithm is shown in Figure 1. First, a

preprocessing step, adapted from [Sung and Poggio, 1994],

is applied to a window of the image. The window is then

passed through a neural network, which decides whether

the window contains a face. The preprocessing ﬁrst at-

tempts to equalize the intensity values across the window.

We ﬁt a function which varies linearly across the window

to the intensity values in an oval region inside the window.

Pixels outside the oval may represent the background, so

those intensityvalues are ignored in computing the lighting

Input

Network Output

subsampling

Preprocessing Neural network

pixels

20 by 20

Extracted windowInput image pyramid (20 by 20 pixels) Correct lighting Histogram equalization Receptive fields Hidden units

Figure 1:

The basic algorithm used for face detection.

variation across the face. The linear function will approx-

imate the overall brightness of each part of the window,

and can be subtracted from the windowto compensate for a

variety of lighting conditions. Then histogram equalization

is performed, which non-linearly maps the intensity values

to expand the range of intensities in the window. The his-

togram is computed for pixels inside an oval region in the

window. This compensates for differences in camera input

gains, and improves the contrast in some cases.

The preprocessed windowis then passed througha neural

network. The network has retinal connections to its input

layer; the receptive ﬁelds of hidden units are shown in

Figure 1. There are three types of hidden units: 4 which

look at 10x10 pixel subregions, 16 which look at 5x5 pixel

subregions, and 6 which look at overlapping 20x5 pixel

horizontalstripes of pixels. Each of these types was chosen

to allow the hidden units to represent localized features that

might be important for face detection. Although the ﬁgure

shows a single hidden unit for each subregion of the input,

these units can be replicated. For the experiments which

are described later, we use networkswith two and threesets

of these hidden units. Similar input connection patterns are

commonly used in speech and character recognition tasks

[Waibel et al., 1989, Le Cun et al., 1989]. The network has

a single, real-valued output, which indicates whether or not

the window contains a face.

Totraintheneuralnetworkusedinstageonetoserveasan

accurate ﬁlter, a large number of face and non-face images

are needed. Nearly 1050 face examples were gathered

from face databases at CMU and Harvard2. The images

contained faces of various sizes, orientations, positions,

and intensities. The eyes and the center of the upper lip

of each face were located manually, and these points were

used to normalize each face to the same scale, orientation,

2Dr. Woodward Yang at Harvard provided over400 mug-shot images.

and position, as follows:

1. Rotate image so both eyes appear on a horizontal line.

2. Scale image so the distance from the point between

the eyes to the upper lip is 12 pixels.

3. Extract a 20x20 pixel region, centered 1 pixel above

the point between the eyes and the upper lip.

In the training set,15 face examples are generated from each

original image,by randomly rotating theimages (abouttheir

center points) up to 10



, scaling between 90% and 110%,

translating up to half a pixel, and mirroring. Each 20x20

windowin theset isthenpreprocessed (byapplyinglighting

correction and histogram equalization). The randomization

gives the ﬁlter invariance to translations of less than a pixel

and scalings of



10%. Larger changes in translation and

scale are dealt with by applying the ﬁlter at every pixel

position in an image pyramid, in which the images are

scaled by factors of 1.2.

Practically any image can serve as a non-face example

because the space of non-face images is much larger than

the space of face images. However, collecting small yet a

“representative” set of non-faces is difﬁcult. Instead of col-

lecting the images before trainingis started, the images are

collected during training in the following manner, adapted

from [Sung and Poggio, 1994]:

1. Create an initial set of non-face images by generating

1000 images with random pixel intensities. Apply the

preprocessing steps to each of these images.

2. Train the neural network to produce an output of 1 for

the face examples, and -1 for the non-face examples.

The training algorithm is standard error backpropoga-

tion. On the ﬁrst iteration of this loop, the network’s

weights are initially random. After the ﬁrst iteration,

we use the weights computed by training in the previ-

ous iteration as the starting point for training.

3. Run the system on an image of scenery which contains

no faces. Collect subimages in which the network

incorrectly identiﬁes a face (an output activation

0).

4. Select up to 250 of these subimages at random, apply

the preprocessing steps, and add them intothe training

set as negative examples. Go to step 2.

We used 120 images of scenery for collecting negative

examples in this bootstrap manner. A typical training

run selects approximately 8000 non-face images from the

146,212,178 subimages that are available at all locations

and scales in the training scenery images.

2.2 Stage two: Merging overlapping detections

and arbitration

The system described so far, using a single neural network,

will have some false detections. Below we mention some

techniquesto reducethese errors; for moredetailsthe reader

is referred to [Rowley et al., 1995].

Because of a small amount of position and scale invari-

ance in the ﬁlter, real faces are often detected at multiple

nearby positions and scales, while false detections only ap-

pear at a single position. By setting a minimum threshold

on the number of detections, many false detections can be

eliminated. A second heuristic arises from the fact that

faces rarely overlap in images. If one detection overlaps

with another, the detection with lower conﬁdence can be

removed.

During training, identical networks with different ran-

dom initial weights will select different sets of negative

examples, develop different biases and hence make differ-

ent mistakes. We can exploit this by arbitrating among

the outputs of multiple networks, for instance signalling a

detection only when two networks agree that there is a face.

3 Experimental results

The system was tested on three large sets of images, which

are completely distinct from the training sets. Test Set A

was collected at CMU, and consists of 42 scanned pho-

tographs, newspaper pictures, images collected from the

World Wide Web, and digitized television pictures. These

images contain 169 frontal views of faces, and require the

networks to examine 22,053,124 20x20 pixel windows.

Test Set B consists of 23 images containing 155 faces

(9,678,084 windows); it was used in [Sung and Poggio,

1994]to measure the accuracy of their system. Test Set

C is similar to Test Set A, but contains some images with

more complex backgrounds and without any faces, to more

accurately measure the false detection rate. It contains 65

images, 183 faces, and 51,368,003 windows.3

Rather than providinga binary output,the neural network

ﬁlters produce real values between 1 and -1, indicating

3The test sets are available at http://www.cs.cmu.edu/˜har/faces.html.

whether or not the input contains a face, respectively. A

threshold value of zero is used during training to select the

negative examples (if the network outputs a value of greater

thanzeroforanyinputfromasceneryimage, itisconsidered

a mistake). Although this value is intuitively reasonable,

by changing this value during testing, we can vary how

conservative the system is. We measured the detection and

false positive rates as the threshold was varied from 1 to

-1. At a threshold of 1, the false detection rate is zero, but

no faces are detected. As the threshold is decreased, the

number of correct detections will increase, but so will the

number of false detections. This tradeoff is illustrated in

Figure 2, which shows the detection rate plotted against the

number of false positives as the threshold is varied, for two

independently trained networks. Since the zero threshold

locations are close to the “knees” of the curves, as can

be seen from the ﬁgure, we used a zero threshold value

throughout testing.

0.75

0.8

0.85

0.9

0.95

1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1 1

Fraction of Faces Detected

False Detections per Windows Examined

ROC Curve for Test Sets A, B, and C

zero

Network 1

Network 2

Figure 2:

The detection rate plotted against false posi-

tives as the detection threshold is varied from -1 to 1, for

two networks. The performance was measured over all

images in Test Sets A, B, and C. Network 1 uses two sets

of the hidden units illustrated in Figure 1, while Network

2 uses three sets. The points lab elled \zero" are the zero

threshold p oints used for all other exp eriments.

Table 1 shows the performance for four networks work-

ing alone, the effect of overlap elimination and collapsing

multiple detections, and the results of using ANDing, OR-

ing, voting,and neural network arbitration. Networks3 and

4 are identical to Networks 1 and 2, respectively, except that

the negative example images were presented in a different

order during training. The results for ANDing and ORing

networks were based on Networks 1 and 2, while voting

was based on Networks 1, 2, and 3. The table shows the

percentage of faces correctly detected, and the number of

false detections over the combination of Test Sets A, B,

and C. [Rowley et al., 1995]gives a breakdown of the per-

formance of each of these system for each of the three test

Table 1:

Combined detection and error rates forTest Sets A, B, and C

Missed Detect False False detect

Type System faces rate detects rate

0) Ideal System 0/507 100.0% 0 0/83099211

Single

network, no

heuristics

1) Network 1 (52 hidden units, 2905 connections) 37 92.7% 1768 1/47002

2) Network 2 (78 hidden units, 4357 connections) 41 91.9% 1546 1/53751

3) Network 3 (52 hidden units, 2905 connections) 44 91.3% 2176 1/38189

4) Network 4 (78 hidden units, 4357 connections) 37 92.7% 2508 1/33134

Single

network,

with

heuristics

5) Network 1

threshold(2,1)

overlap elimination 46 90.9% 844 1/98459

6) Network 2

threshold(2,1)

overlap elimination 53 89.5% 719 1/115576

7) Network 3

threshold(2,1)

overlap elimination 53 89.5% 975 1/85230

8) Network 4

threshold(2,1)

overlap elimination 47 90.7% 1052 1/78992

Arbitrating

among two

networks

9) Networks 1 and 2

AND(0) 66 87.0% 209 1/397604

10) Networks 1 and 2

AND(0)

threshold(2,3)

overlap elimination 107 78.9% 8 1/10387401

11) Networks 1 and 2

threshold(2,2)

overlap elimination

AND(2) 74 85.4% 63 1/1319035

12) Networks1 and 2

thresh(2,2)

overlap

OR(2)

thresh(2,1)

overlap 48 90.5% 362 1/229556

Three nets 13) Networks 1, 2, 3

voting(0)

overlap elimination 53 89.5% 195 1/426150

threshold(distance,threshold): Only accept a detection if there are at least threshold detections within a cube (extending along x, y, and scale) in the

detection pyramid surrounding the detection. The size of the cube is determined by distance, which is the number of a pixels from the center of the

cube to its edge (in either position or scale).

overlap elimination: A set of detections may erroneously indicate that some faces overlap with one another. This heuristic examines detections in order

(from those having the most votes within a small neighborhoodto those having the least), and removingconﬂicting overlapsas it goes.

voting(distance), AND(distance), OR(distance): These heuristics are used for arbitrating among multiple networks. They take a distance parameter,

similar to that used by the threshold heuristic, which indicates how close detections from individualnetworks must be to one anotherto be counted as

occuring at the same location and scale. A distance of zero indicates that the detections must occur at precisely the same location and scale. Voting

requires two out of three networks to detect a face,AND requires two out of two, and OR requires one out of two to signal a detection.

sets, as well as the performance of systems using neural

networksto arbitrationamong multipledetection networks.

The parameters required for each arbitration method are

described below the table.

Systems 1 through 4 show the raw performance of the

networks. Systems 5 through 8 use the same networks,

but include the thresholding and overlap elimination steps

which decrease the number of false detections signiﬁcantly,

at the expense of a small decrease in the detection rate. The

remaining systems all use arbitration among multiple net-

works. Arbitration further reduces the false positive rate,

and in some cases increases the detection rate slightly. Note

that for systems using arbitration, the ratio of false detec-

tions to windows examined is extremely low, ranging from

1 false detection per 229,556 windows to down to 1 in

10,387,401, depending on the type of arbitrationused. Sys-

tems 10, 11, and 12 show that the detector can be tuned to

make it more or less conservative. System 10, which uses

ANDing, gives an extremely small number of false posi-

tives, and has a detection rate of about 78.9%. On the other

hand, System 12, which is based on ORing, has a higher

detection rate of90.5% butalso has a larger number of false

detections. System 11 provides a compromise between the

two. The differences in performance of these systems can

beunderstoodby consideringthearbitrationstrategy. When

using ANDing, a false detection made by only one network

is suppressed, leading to a lower false positive rate. On the

other hand, when ORing is used, faces detected correctly by

only one network will be preserved,improving thedetection

rate. System 13, which votes among three networks, yields

about the same detection rate and lower false positive rate

than System 12, which using ORing with two networks.

Based on theresults inTable 1, we concluded that System

11 makes an reasonable tradeoff between the number of

false detections and the detection rate. System 11 detects

on average 85.4% of the faces, with an average of one false

detection per 1,319,035 20x20 pixel windows examined.

Figure 3 shows examples output images from System 11.

4 Comparison to other systems

[Sung and Poggio, 1994]reports a face detection system

based on clustering techniques. Their system, like ours,

passes a small window over all portions of the image, and

determines whether a face exists in each window. Their

system uses a supervised clustering method with six “face”

and six “non-face” clusters. Two distance metrics measure

the distance of an input image to the prototype clusters.

The ﬁrst metric measures the “partial” distance between the

test pattern and the cluster’s 75 most signiﬁcant eigenvec-

tors. The second distance metric is the Euclidean distance

between the test pattern and its projection in the 75 dimen-

sional subspace. These distance measures have close ties

with Principal Components Analysis (PCA), as described

in [Sung and Poggio, 1994]. The last step in their system is

to use either a perceptron or a neural network with a hidden

D: 3/3/0

P: 1/1/0

E: 1/1/0

C: 1/1/0

N: 8/5/0

A: 12/11/3

M: 1/1/0

B: 6/5/1

O: 1/1/0

L: 4/4/0

K: 1/1/0

J: 1/1/0

T: 1/1/0

S: 1/1/0

I: 1/1/0

F: 1/1/0

Q: 1/1/0

G: 2/2/0 H: 4/3/0

R: 1/1/0

Figure 3:

Output obtained from System 11 in Table 1. For each image, three numb ers are shown: the numb er of faces

in the image, the number of faces detected correctly, and the numb er of false detections. Faces are missed in A and H

(for unknown reasons), B (large angle), and N (the stylized faces are not reliably detected at the same locations and

scales by the two networks, and so are lost by the AND heuristic). False detections are present in A and B. Although the

system was trained only on real faces, hand drawn faces are detected in I and N. Images A, H, K, and R were scanned

from printed photographs, B, D, G, I, L, and P were obtained from the World Wide Web, C, E, and S are digitized

television images, F, J, M, and Q were scanned from photographs, N and T were provided by Sung and Poggio at MIT,

and O is a dithered CCD image.

layer, trained to classify points using the two distances to

each of the clusters (a total of 24 inputs). The main com-

putational cost in [Sung and Poggio, 1994]is in computing

the two distance measures from each new window to 12

clusters. We estimate that this computation requires ﬁfty

times as many ﬂoating point operations as are needed to

classify a window in our system, in which the main costs

are in preprocessing and applying neural networks to the

window. Table 2 shows the accuracy of their system on

Test Set B, along with the our results using Systems 10, 11,

and 12 in Table 1, and shows that for equal numbers of false

detections, we can achieve higher detection rates.

Table 2:

Comparison of

[

Sung and Poggio, 1994

]

and our system on Test Set B

Missed Detect False False detect

System faces rate detects rate

10) Networks 1 and 2

AND(0)

threshold(2,3)

overlap elimination 34 78.1% 3 1/3226028

11) Networks 1 and 2

threshold(2,2)

overlap elimination

AND(2) 20 87.1% 15 1/645206

12) Networks 1 and 2

threshold(2,2)

overlap

OR(2)

threshold(2,1)

overlap 11 92.9% 64 1/151220

[Sung and Poggio, 1994](Multi-layer network) 36 76.8% 5 1/1929655

[Sung and Poggio, 1994](Perceptron) 28 81.9% 13 1/742175

Although our system is less computationally expensive

than [Sung and Poggio, 1994], the system described so far

is not real-time because of the number of windows which

must be classiﬁed. In the related task of license plate detec-

tion, [Umezaki, 1995]decreased the number of windows

that must be processed. The key idea was to have the neural

networkbe invariant to translations of about 25% of the size

of a license plate. Instead of a single number indicatingthe

existence of a face in the window, the output of Umezaki’s

network is an image with a peak indicating where the net-

work believes a license plate is located. These outputs are

accumulated over the entire image, and peaks are extracted

to give candidate locations for license plates. In [Rowley et

al., 1995], we show that a face detection network can also

be made translation invariant. However, this translation in-

variant face detector makes many more false detections than

one that detects only centered faces. We use the centered

face detector to verify candidates found by the translation

invariant network. With this approach, we can process a

320x240pixel image in less than 5 seconds on an SGI Indy

workstation. This technique is related, at a high level, to

the technique presented in [Vaillant et al., 1994].

5 Conclusions and future research

Our algorithm can detect between 78.9% and 90.5% of faces

in a set of 130 test images, with an acceptable number of

false detections. Depending on the application, the system

can be made more or less conservative by varying the arbi-

tration heuristics or thresholds used. The system has been

tested on a wide variety of images, with many faces and

unconstrained backgrounds.

There are a number of directions for future work. The

main limitation of the current system is that it only detects

upright faces looking at the camera. Separate versions of

the system could be trained for different head orientations,

andthe resultscould be combined usingarbitrationmethods

similar to those presented here. Other methods of improv-

ingsystem performance includeobtainingmorepositiveex-

amples for training, or applying more sophisticated image

preprocessing and normalization techniques. For instance,

the color segmentation method used in [Hunke, 1994]for

color-based face tracking could be used to ﬁlter images.

The face detector would then be appliedonly to portionsof

the image which contain skin color, which would speed up

the algorithm as well as eliminating some false detections.

One application of this work is in the area of media tech-

nology. Every year, improved technology provides cheaper

and more efﬁcient ways of storing information. However,

automatic high-level classiﬁcation of the information con-

tent is very limited; this is a bottleneck preventing media

technology from reaching its full potential. The work de-

scribed above allows a user to make queries of the form

“Which scenes in this video contain human faces?” and to

have the query answered automatically.

Acknowledgements

The authors thank Kah-Kay Sung and Dr. Tomaso Pog-

gio (at MIT) and Dr. Woodward Yang (at Harvard) for

providing a series of test images and a mug-shot database,

respectively. Michael Smith (at CMU) provided some digi-

tized television images for testing purposes. We also thank

Eugene Fink, Xue-Mei Wang, Hao-ChiWong, TimRowley,

and Kaari Flagstad for comments on drafts of this paper.

References

[Hunke, 1994]H. Martin Hunke. Locating and tracking of hu-

man faces with neural networks. Master’s thesis, University of

Karlsruhe, 1994.

[Le Cun et al., 1989]Y. Le Cun, B. Boser, J. S. Denker, D. Hen-

derson,R. E. Howard,W. Hubbard,and L. D. Jackel. Backpro-

pogation applied to handwritten zip code recognition. Neural

Computation, 1:541–551, 1989.

[Rowley et al., 1995]Henry A. Rowley, Shumeet Baluja, and

Takeo Kanade. Human face detection in visual scenes. CMU-

CS-95-158R, Carnegie Mellon University, November 1995.

Also available at http://www.cs.cmu.edu/˜har/faces.html.

[Sung and Poggio, 1994]Kah-Kay Sung and Tomaso Poggio.

Example-basedlearning for view-based human face detection.

A.I. Memo 1521, CBCL Paper 112, MIT, December 1994.

[Umezaki, 1995]Tazio Umezaki. Personal communication,

1995.

[Vaillant et al., 1994]R. Vaillant, C. Monrocq, and Y. Le Cun.

Original approach for the localisation of objects in images. IEE

Proceedings on Vision, Image, and Signal Processing, 141(4),

August 1994.

[Waibel et al., 1989]Alex Waibel, Toshiyuki Hanazawa, Geof-

frey Hinton, Kiyohiro Shikano, and Kevin J. Lang. Phoneme

recognition using time-delay neural networks. Readings in

Speech Recognition, pages 393–404, 1989.